Hyperplane Dynamics as a Means to Understanding Back-Propagation Learning and Network Plasticity

The processing performed by a feed-forward neural network is oft en int erpreted through use of decision hyperplanes at each layer. T he adaptation process, however, is normally explained using the picture of gradient descent of an error land scape. In thi s paper t he dynamics of t he decision hyperplanes is used as t he model of the adaptat ion process. An electro-mechanical analogy is drawn where t he dynamics of hyperplanes is determined by interaction forces between hyperplanes and the particles that represent t he patterns. Relaxati on of the system is determined by increasing hyperplane inerti a (mass). This picture is used to clar ify the dynamics of learning, and goes some way toward explaining learning deadlocks and escaping from certain local minima. Furthermore, network plast icity is introduced as a dynamic property of the system, and reduction of plasticity as a necessary consequence of informat ion storage . Hyperplane inertia is used to explain and avoid destructive relearning in t rained networks.